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ABSTRACT 

A field study tested the application of the CONSULT-I 
(R) program, which uses artificial intelligence with statistical 
pattern recognition in constructing a diagnosis and recommending 
treatment of reading difficulties. Participants in the field study 
came from 10 southern and central Indiana school districts, both 
public and parochial, and one Massachusetts school district. 
Participants included one high school dean in charge of the tutoring 
program, five teachers working with middle school learners, and 21 
elementary school personnel. A total of 70 learners were taught by 
the 23 teachers who eventually completed the required data 
collection. Teachers completed learner profile sheets, carried out 
the recommendations of the CONSULT- I (R) program, and completed 
questionnaires at the end of the 5-month implementation period. 
Results indicated a positive reaction to most of the CONSULT-I (R) 
program: the focus in individualization and the strategies suggested 
were judged to be very helpful; and the negative responses showed a 
need for more written guidance in several aspects of the program. 
Findings suggest that data based diagnostic and treatment 
recommendations do work, and that through the use of the program, 
educators are able to provide an answer to the inconsistency of 
treatment predications for classroom teachers and reading 
specialists. Recommendations include: continue trials of the program 
in graduate and undergraduate classes; offer the program to area 
schools on a fee basis; and provide more written instructions. 
(Contains four tables of data,) (RS) 
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Introduction 

One of the unsolved problems in the field of reading is the 
inconsistency of diagnostic predictions and treatment 
prescriptions among reading teachers and specialists. In a 
series of studies conducted by Vi nsonhal er ? WeinshanL , Wagner and 
Pqlin (1987.)* it was found that 

Mean diagnostic agreement between two clinicians 
remained close to 0.10 across the six studies. 
Me*n diagnostic agreement for a single clinician 
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agreement were similar to those for diagnosis. • r 
Further, remediation^ appeared to be uncorrelated 
with diagnosis <p. 134> . 
Advanced technology now offers the possibility of a solution to 
this problem as seen in the following report of research 
conducted by Indiana University's Reading Practicum Center. 
Hi story 

A unique opportunity was offered to the Language Education 
Department of the School of Education, Indiana University in 1983 
through the friendship oi Nicholas Fattu and Leo Fay. Outcome 
Advisor (R) and CONSULT-I (R) , computer programs designed by 
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Edward Patrick, M.D., Ph.D. and James Fattu, M.D., Ph.D. (son of 
Nicholas Fi.ttu) -for use in medical diagnosis and prescription, 
were given to the Language Education Department for development 
in the field of education. Anabel Newman, director of the 
Reading Practicum Center, was asked to guide the development. 

Outcome Advisor <R> and CONSULT-I <R) have not only been 
successful in the field of medicine for diagnosis and 
prescription, but also in the fields of agriculture, geology and 
business. Both programs are expert systems requiring expert 
judgment on the part of the user. The data base is entered in 
Outcome Advisor <R>, probability densities are constructed -from 
the data base within the program, and then this information is 
used to train CONSULT- I (R) for diagnosis and prescription. Both 
programs use artificial intelligence with statistical pattern 
recognition in constructing outcomes (see Artificial Intelligence 
with Statistical Pattern Recogrut i_gn , Patrick ?< Fattu, 1986). 

To determine the potential of the programs in education, a 
pilot study was conducted to demonstrate the ability of Outcome 
Advisor <R) to predict success as well as or better than experts 
in the field of reading. Standardized test scores of low 
readiness first graders were used to predict success at the sixth 
grade level (data came from longitudinal studies previously 
conducted by Newman, 1978, 1980, 1985). Outcome Advisor <R) 
predicted success slightly better than the experts (627. to 59V.). 
This result seemed to justify further exploration. 

One of the intriguing aspects of Outcome Advisor <R) and 
CONSULT-I <R> is their ability to take into consideration a wide 



range of characteristics. This matched with the prulosophv of 
the Reading Pr act* cum Center <RPC) personnel who -feel that the 
whole learner (cultural, physiological, psycho! ogi cal , and 
educational characteristics) must be taken into consideration 
before decisions for instructional strategies can be made. A 
second promising aspect was that the program, although requiring 
a criterion measure, did not require standardized scores to judge 
success. This also matched the RPC philosophy as pre and post 
standardized tests are not usually given to a learner. Rather, 
success is measured through changes in affect and attitude, and 
movement toward or attainment of goals. These criteria are used 
because after /ears of observation by RPC personnel, it is felt 
that these changes come before changes in standardized test 
scores. Descriptions of learner changes in affect, attitude, and 
success were taken from actual case studies and categorized into 
high, medium and low* These were 

then used as criteria for determining the amount of change in any 
individual learner. 



The next step in the process was to develop a taxonomy of 
learner characteristics drawn from the research literature plus 
personal experience which might discriminate between those 
learners who would be highly successful in a reading program 
similar to that of the RPC and those who would not. The original 
taxonomy — a list of 92 features each with 4 to 9 values 

— was developed over a period of several months 
through a careful process of categorization using logical 



inference to determine features and values. Included were 
conventional characteristics such as socioeconomic status, 
placement among siblings, age, and being read to before starting 
school as well as items such as cultural motivation (working, 
playing with parents), model, interests, and pressure — 
variables positively correlated with reading success as 
i denti f 1 ed in Newman ' s 1 ongi tudi nal studi es. 

To create the data base, past case studies (from 1975 to 
1985) on file at the KPC were read and learner profiles 

coded according to the taxonomy. Case studies from 
1985 to 1988 were added later. These data were entered into 
Outcome Advisor <R> and probability densities were run. As the 
data base grew, it became evident that some of the features were 
not discriminating between high and low success, so they were 
periodically eliminated until the present taxonomy of 23 features 

was reached with a data base of 183 learners. 
There are now 218 learners included in the data base. 

Before CONSULT-I (R) could be trained, however, it was 
necessary to match successful instructional strategies with 
individual learners in the data base. The same two readers again 
read the 183 case studies to determine the one strategy which, in 
their judgment, had made the difference in each learner's 
success. Eight categories of strategies emerged! comprehension, 
functional language, gaming, interests, language experience, 
motivation, self -concept enhancement, and study skills. (While 
other strategies such as phonics and drills were used, they never 
appeared to be the turning point in a learner's experience so 



are not included in the eight categories.) This most *ucc*ssful 
instructional strategy was then added to each profile, entered in 
Outcome Advisor (R) and probability densities run. The 
information from the probability densities was then used to train 
CONSULT-I (R) which now could be used to recommend instructional 
strategies for given learners. 

In continuing studies students in Newman's reading methods 
classes (undergraduate), as well as the diagnosis classes and 
language practicums (graduate) and Met 2 51 diagnosis classes and 
language practicums (graduate) were asked to complete profiles on 
each of their learners (1986 - 19S9) . The profiles were entered 
into CONSULT-I (R) and the recommendations made by the program 
were returned to the students. In over 95V. of the cases it was 
felt by the students that the recommendations made by CONSULT-I 
(R) were on target. 

During the spring of 19S9 the students in both the 
undergraduate reading methods class and the graduate practicum 
were asled to track learner changes in affect, attitude and 
success throughout the semester* 

Their use of CONSULT-I (R) recommended strategies was also 
monitored through monthly observations and interviews. 

In addition to the studies within the university classes, 
CONSULT-I (R) was used by the Institute for Child Study's 
Interdisciplinary Clinic in cases involving reading problems. 
Parents, calling the Reading Practicum Center for help with their 
child's reading problems, were also given the opportunity to 
receive recommendations from CONSULT-I (R) . Again, the 
recommendations made by CONSULT-I (R) were judged by users to be 

5 



appropriate. 

At the same time as the above studies were being conducted, 
a booklet < Individualizing Language Strategies Usi^ng CONSULT-i 
(R) ? Newman & Metz , 1988) w'<ich would describe the various 
categories of instructional strategies was under development. 

The strategies contained in the booklet were drawn 
-from one or more o-f the cases contained in the data base and are 
intended to suggest successful directions for instructions for 
learners with certain characteristics. Completed during the 
field study described below, the booklet is to be used as a base 
for an individualized program built around the recommendations 
of CONSULT-I (R> and the learner's interests. 

Given the successful trial of the program in these studies, 
it was decided to field test the program in a cross section of 
area schools. A Proffitt. Grant was applied for and received by 
Newman to conduct the stud/ over the course of a year. 
Fi el_d Study 

Participants. In May of 1988 several area school districts 
were contacted about participation in the study. All of them 
felt a decision would need to be made in the fall after school 
had started. Several individuals did contact Newman at this time 
in regards to updating class work needed for graduate degrees. 
A decision was made for them to accomplish this through 
participation in the field study. 

School districts were agai n contacted in the fall and 
several meetings to present CONSULT-I <R) to teachers were set 
up. At the same time the opportunity to participate was offered 



through flyers to individual schools and district newsletters* 

(There was no response to these.) In 
November the opportunity to participate was once again offered 
through a presentation on CONSULT-I (R) at the Indiana 
University's Fall Language Arts Conference. 

During these preliminary meetings with teachers, individual 
packets containing a cop/ of the taxonomy, a learner profile 
sheet, a sample computer printout of CONSULT-I <R) 
recommendations, and background information on CONSULT-I (R) were 
distributed • the background of the program was 

presented; the features of the taxonomy were discussed; and the 
expectations of participation in the field study were outlined. 
These expectations included collecting learner data, carrying out 
the recommendations over a period of time and completing 
evaluation forms. The depth of discussion varied according to 
the amount of time (15 minutes to 45 minutes) allowed by the 
group for the presentation. 

Although much interest was in evidence at the various 
meetings, it was difficult to find teachers who were willing or 
felt their situations would allow them to follow through and 
become actual participants. For example, at one meeting with 
Chapter One teachers, seven volunteered to participate. But, 
when it came time to collect the learner data, all decided that 
the program was in conflict with the Chapter One requirements. 
Three Chapter One teachers in another district did participate. 
At the Fall Language Arts Conference twenty three teachers 
volunteered to participate, but only four actually collected the 
data and carried out the program. Reasons given for non- 
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participation included Uck of time due to other demands such as 
a North Central evaluation and an unwillingness to ask parents 
-for some of the in* ormation. 

Participants in the field study came from ten southern and 
central Indiana school districts, both public and parochial, and 
one Massachusetts school district. They included one high school 
dean in charge of the school tutoring program, five teachers 
working with middle school learners, and twenty-one elementary 
school personnel. Both, rural and urban areas were represented. 

Data collection and processing . Gradually throughout the 
fall, despite the various obstacles, twenty-three teachers did 
collect the data and returned learner profile sheets to the 
Reading Practicum Center. The number of learners for each 
teacher varied from one to seven wxth an average of 3 and a total 
of 70. As the profile sheets came into the Center the data were 
processed through CONSULT-I <R). Recommendations made by the 
program were noted on the profile sheet • 

The profile sheets with the CONSULT-I <R> 
recommendations; a letter of explanation 

and the booklet, Individualizing L£QQU£9§ 
strategies using QONSULT-I JRi, were sent to the teacher. In a 
few cases the recommendations and booklet were returned in person 
by one of the researchers with a short oral explanation of the 
recommend at i ons. 

AgQiicatign of cecgmmendat i gns. The teachers, at both the 
preliminary meeting and in the Letter, were asked to c^rry out 
the recommendations from October through February. Originally 
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scheduled to end in February, the instructional time was extended 
due to the varying entry times of the teachers. 

§*£Ly*t i.on. Questionnaires were developed during the spring 
to be sent to the teachers at the end of the study for evaluation 
of the study and the progress made by the learners as evidenced 
by changes in affect and attitude and success (or progress) in 
the classroom* Directions for completion of the learner 

evaluation were included. 

The teachers were asked to complete the forms 
within one week and return them to the Reading Practicum Center. 

Results. The teacher evaluations of the program are shown 
in Table 1- Totals of the top three rankings yielded the 
following results: 

tlOOV. agreed that the introduction to the program was easily 
understood - 

♦ 1007. agreed that the taxonomy was easily understood. 
»aav. agreed that the profile sheet was easily understood. 
*3B7. agreed that the collection of learner data was 
ef f i c 1 ent . 

*1007. agreed that the recommendations for individual 

learners seemed appropriate. 
*1007. agreed that the strategy booklet's explanations and 

examples were easily understood. 
*897. agreed that the strategy booklet was useful for program 

devel opment . 

$837. agreed * that it was easy to implement the 

recommend at l ons. 
$937. agreed that the recommendations were practical for 
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cl assroom use. 

*797. agreed that the support /mom tor 1 ng by RFC staff was 
hel pf ul . 

The areas which included negative responses were: 

*127> disagreed that the pro-file sheet was easily understood* 
*127. disagreed that the collection of learner data was 
ef f i ci ent . 

*117, disagreed that the. strategy booklet was useful for 
program development. 

* 1 77. di sagreed that it was easy to 1 mpl ement the 

recommend at i ons. 

■> 

*77. disagreed that the recommendations were practical for 
classroom use. 

♦ 217. disagreed that the support /moni tor i ng by RPC staff was 

hel pf ul . 

Comments pertaining to the best part and the weakest part of 
the CONSULT-I (R) program are shown in Table 2. According to the 
responding teachers, the strategy booklet with its easy to use, 
practical ideas was the best part of the program. Other items 
mentioned included being forced to focus on the individual 
learner, the recommendations, the support for the classroom 
teacher, the variety and flexibility, and the individualized 
approach. As one teacher characterized her experience, the best 
part of CONSULT-I (R) is "making me look closely at these 5 
students 1 We developed more of a relationship from filling out 
forms •together.' I solicited answers from parents on some 
questions - involving them , too' I'll definitely use (an) 
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interest inventory in the -future. My students succeeded because 
of improved seH-concept which I could encourage -from the support 
of your program. I am noticing reading grew the most 1 I wonder 
if writing will come as they enjoy reading to a greater extent*?" 

The weakest parts of the program as seen by the teachers were 
the collection of data for the taxonomy and implementing the 
strategies in the classroom. 

The learner evaluations (see Tables 3 and 4) concerned with 
changes in affect showed that: 

*737. now enjoy reading more 

*797. now choose to read more 

*487. now enjoy writing more 

*417. now choose to write more. 

Changes in attitude (see Tables 3 and 4) according to the 
evaluations showed 

*577. have a better attitude toward learning now 
*53% have a better self image. 

High success as evidenced by significant progress in the 
classroom was demonstrated by 317. of the learners. Fifty-eight 
percent of the learners showed medium success (progress in the 
classroom) while 117. made little or no progress (low success). 
Summary 

The results of the study, as reflected in the program 
evaluations, show a positive reaction to most of the CQNSULT-I 
(R) program. The focus on individualization and the strategies 
suggested in the booklet, Ind LwduaH r i_i.ng Language Strategies 
L)si_ng CQNSULI^I JR)_, were judged by the users to be very helpful. 
The negative responses show a need for more written guidance in 
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several aspects of the program: taxonomy, profile sheet, and 

Qroup implementation. 

Learner evaluation reflected changes an affect in reading in 
* large number of learners and in writing in a moderate number of 
learners. The difference between the results in eading and 
those in writing show the need for more emplasis on writing 
strategies. There were also a moderate number of learners who 
changed in attitude toward learning and in self-image. The 
smaller number of learners with change in attitude complements 
the previous observations of the RFC staff that affect usually 
changes before attitude. 
Di,£CLiS§ v_on 

The results of this study suggest that data based diagnostic 

and treatment recommendations do work, and that through the use 

of the CONSULT-I <R) program we are now able to provide an answer 

to the inconsistency of treatment predictions for classroom 

teachers and reading specialists. Feedback on positive aspects 

of the program was provided as well as identification of areas 

where further clarification is needed. 

Re^om^endat -— 

Given the findings, we recommend the following: 

1. Continue trials of CONSULT-I (R) with graduate and 
undergraduate classes. Prepare teachers for the 
possibility of implementing recommendations for 
individual learners. 

2. Offer CONSULT-I (R) to area schools on a fee basis. 

3. Provide more complete written instructions including 
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*a taxonomy guide 'for responding to taxonomy choices 
♦ written directions for compl et 1 ng the prof lie sheet 
fsuggestions in the strategy booklet for implementing 
recommend at 1 ons for group instruction. 
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Table 1. Teacher Evaluation of Program - Likert Scale Responses 



Strongly Strongly 
Agree Disagree 
12 3 4 5 



K The introduction to CONSULT- 1 (R> 447. 507. 67. 
was easily understood. 



2. The taxonomy was easily 
under stood . 
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3. The profile sheet was easily 357. 357. 187. 127. 
understood . 

4. The method of collection of 237. 337. 277. 67. 67. 
learner data was efficient. 

5. The recommendations for individual 39/1 55/. 67. 
learners seemed appropriate. 

6. The strategy booklet's explan- 667. 287. 67. 
ations and examples were easily 

understood . 

7. The strategy booklet was useful 557. 287. 67. 117. 
for program development. 

8. It was easy to implement the 447. 397. 177. 
r ecommendat i ons . 

9. The recommendations were prac- 537. 277. 137 77. 
tical for classroom use. 

10. The support/monitoring by 297. 217. 297. 14% 77. 
RFC staff was helpful. 
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Table 2. Teacher Evaluation of Program - Anecdotal Responses 

Best Part 

Strategy booklet 

Strategy book - wealth of practical, innovative ideas for 
enhanci ng curr i cul urn 

Provided many easy to use ideas, very practical 

Ideas for what to do with student to increase interest in 
readi ng 

Practical and useful projects to encourage reading and 
writing 

Easy to use, efficient, material targeted well 

Pinpointing- of specific problem(s> and focusing on problem 
area to build with specific tactics 

Rewarding to know on the right track, forced to evaluate all 
aspects of learner's environment, understood needs and 
mot i vat i ons 

Recommendat 1 ons 

Provides more support for classroom teacher 
Var lety and flexibility 

Individualized approach, addressing affect as precursor to 
reading acti vi ties 

Weakest Part 

Difficult to collect d,ata for taxonomy 

Col 1 ect i ng 1 earner data 

Some o-f the items on taxonomy list were con-fusing 
Compl et i ng prof i 1 e sheet . 
Implementation in classroom environment 
Chapter One limitations 

Difficulty in implementing program in classroom without 
training 

Most at the methods for improvement were techniques already 
empl oyed in cl assroom 

Am 



Table 3. Learner Placement Pre and Post On Affect and Attitude* 



Low 



Medi urn 



Hxgh 



Enjoys reading 



>: x x x xoooo 

>: >: x x >! 
x x x x x 
x x x x x 
x x x x x 
x x x x x 
x x 



X X X X X ooooo 
xxxxxooooo 
X X X X X ooooo 
X X XXX ooooo 
X ooooo 
oooo 



X X ooooo 
ooooo 
ooooo 
ooooo 
oo 



Chooses to read 



X X X XX oooo 
>: x x x x 
x x x x x 

X X X X X 
X X X X X 
X X X X X 
X X X X X 



XX XXX ooooo 
X X X X X ooooo 
X X X X X ooooo 
X X ooooo 
ooooo 
oooo 



X ooooo 
ooooo 
ooooo 
ooooo 



Enjoys writing 



>: x x x x ooooo 
XXX X X ooooo 
X X XX xoooo 

x x x x x 

X X X X X 
X X X X X 
X X X X X 
X X X 



X X X X X ooooo 
>: x x x x ooooo 
X X X X ooooo 
ooooo 
ooooo 
ooooo 
o 



ooooo 
oo 



Chaoses to write 



x x x >: x ooooo 
x x x x x ooooo 
x x x >: >: ooooo 
X X X X X ooooo 

x x x >: >: o 
x x x x x 

X X X X X 

x x x >: x 
x 



x x x >: >: ooooo 

X >! X >! X OOOOO 
X OOOOO 

ooooo 
ooooo 



ooooo 
o 



Attitude to learning 



xxxx x ooooo 

X X X X X o 
X X X X X 
X X X X X 
X X X >! X 
X X X 



XXXX X ooooo 
XX xxxooooo 
xxxxxooooo 
X X X X X ooooo 
xxxooooo 
ooooo 
ooo 



X X X X ooooo 
ooooo 
ooooo 
o 



Sel f -i mage 



xxxxxooooo 

X X X X X oo 
XXX XX 

XX xxx 
X X X X X 
X 



xxxxxooooo 
xxxxxooooo 
XX X XX ooooo 
X xxxx ooooo 
X XX ooooo 
ooooo 



xxxooooo 
ooooo 
ooooo 



« Pre, o Post 



♦Not all teachers responded to all items. 
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Table 4. Changes in Affect and Attitude Over the Course of the 
Field Study (1) (Pre - Post), 



L 


- L 


L - M 


L - H 


M - L 


M - M 


M - H 


H - h 


Enjoys reading 


4 


20 


8 




9 


12 


2 


Chooses to read 


4 


23 


8 




6 


1 1 


1 


Enjoys writing 


14 


18 


6 




13 


1 




Chooses to write 


21 


15 


4 




9 


2 




Attitude - learning 


er 

vJ 


20 


4 


1 


13 


8 


5 


Sel f -l mage 


5 


16 


5 


2 


15 


7 





L — Low 

M = Medium 

H = High 



(1) Four to seven months depending on time of entry into study 
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